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ABSTRACT 

As part of an effort to improve students’ knowledge of constellations and bright stars in an 
introductory level descriptive astronomy survey course, we measured the baseline knowledge that 
students bring to the class and how their score evolve over the course of the semester. This 
baseline is needed by the broader astronomy education research community for future 
comparisons about which strategies and environments are the best for learning the stars and 
constellations. As a comparison group, we also examined the baseline knowledge of 14-15 year 
old, 9 th grade students from the United States. 664 university students averaged 2.04±0.08 on a 
constellation knowledge survey, while 46 additional students averaged higher at 8.23±0.23. The 
large, lower scoring group is found to have the same knowledge level as the 14-15 year old 9 th 
grade students which scored 1.79±0.13. The constellations most often identified correctly were 
Orion and Ursa Major. For the star portion of the survey, which was only given to the university 
students, we found essentially no statistically significant prior knowledge for the 17 brightest stars 
surveyed. The average score for the stars was 1.05±0.05, as expected for guessing, although 
Polaris and Betelgeuse are labeled correctly more often than any other stars. 
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n the modem world of education the use of technology is becoming an ever increasing part of teaching. 
A modern planetarium can be seen as one of these technologies. However, the amount of research on 
the effectiveness of using planetarium technology rarely appears in a search of recent literature (c.f, 
Reed 1970; Ridky, 1974; Sunal, 1976). A summary of more recent astronomy education research was conducted by 
Bailey and Slater (2004) who only made a scant few references to early planetarium research. The most interesting 
article they noted, as it relates to planetarium research, is the work of Mallon and Bmce (1982), astronomy 
education researchers who studied how well students learned constellations in both a traditional ‘star show’ 
approach and in a more participatory approach. They found that the participatory approach led to better results as 
would be expected. Their work was focused on students in the age group of 8-10 years old. 


Recent years have seen a number of new efforts to examine the effectiveness of planetariums for teaching. 
Smolinski (2014) tested the transference of information learned in a planetarium to the real night sky. Plummer 
(2009) demonstrated that early elementary school children are able to grasp the motions in the sky when taught in a 
planetarium setting. Later work by Plummer and colleagues (2014) compared the roles of classroom and 
planetarium based instruction finding that planetarium interaction was better. Jacobson (2013) reported on the use of 
a planetarium to teach history in an immersive dome environment and found improved results. Brazzeli and 
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Espinoza (2009) used a meta-analysis of current research on planetarium efficacy research and found planetariums 
to be a useful teaching tool. Building on these studies Zimmerman and colleagues (2014) showed that a portable- 
dome system was just as effective for introducing new subjects as a computer screen, and even better when hoping 
for long term retention. Turk and Kalkan (2015) examined the teaching of material related to the solar system to a 
group of 12-13 year olds. This compared typical classroom instruction to the planetarium environment. They found 
that those students who were taught in the planetarium did better on 3D tasks, comprehending reference systems, 
and on items requiring changing time or periodic motion. Eriksson e and colleagues (2014) also examined 
understanding the 3D nature of the Universe, although not in a full planetarium environment. Finally we have the 
very recent work of Yu and colleagues (2015) where they examined the impact of teach the seasons using a digital 
planetarium. However, few of the projects detailed above directly addressed fundamental constellations and bright 
stars. 


For more than 30 years students in the introductory descriptive astronomy survey course for non-science 
majoring undergraduates—known widely in the United States as “ASTRO 1010”— at Brigham Young University 
(BYU) have been tasked to learn the names and locations of a group of constellations, bright stars, and deep sky 
objects that can be seen in the planetarium theater on campus. While this is largely a task of memorization it is 
considered an important part of the class and is the one common requirement for all sections. This portion of the 
class receives almost universally positive comments on end-of-course student reviews of the class. Anecdotally, 
former students are encountered 10 or more years after taking the class report still remembering the constellations as 
a result of this planetarium-based learning experience. Prior to this project, we had no definitive idea how many 
constellations and bright stars the university students knew prior to taking the class. Understanding incoming 
students’ prior-knowledge is a fundamental underpinning of a constructivist approach to instruction (Slater, 1993). 

The work described here has its origin in two separate, but intertwined, lines of astronomy education 
research. The first line is based on our long-term desire to examine the impact of using different planetarium 
technologies to teach constellations and bright stars. There are a wide range of projector systems that can be used in 
a planetarium. Some generate a very accurate star field, but don’t allow any additional projection. While other 
systems, with a less accurate star field, allow planetarium educators to include stick figures, names, or borders 
projected along with the stars. The long term goal of this research is to determine which of the projector systems is 
the best for learning the stars and constellations, retaining that information, and transferring the knowledge to the 
real sky. However, to provide a fair basis for comparison we need to know the baseline level for incoming 
undergraduate students. A search of the literature yielded no information on a baseline level. This then motivates us 
to pursue two first-steps questions: do students bring any prior knowledge of constellations and bright stars into the 
class, and which constellations and bright stars are the most likely to already be known? Since we were also able to 
track scores over a semester we can also approach answering the question how many instructional sessions— 
hereafter called “labs”—does it take on average for students to reach a given level of knowledge mastery, such as 
90%? The data generated through this systematic effort can help answer that question more reliably. 

A second line of research deals with research into deaf education in a planetarium as summarized in Jones 
and colleagues (2014) and Hintz and colleagues (2015). Those papers demonstrated that hearing-impaired, deaf 
students using head-mounted displays (HMD) are able to adopt American Sign Language (ASL) classifiers used in a 
planetarium show. However, that study was unable to provide strong evidence on actual levels of learning achieved. 
The names of constellations seem to serve as a very clean test set to examine the effectiveness of the HMD in a 
teaching environment. As detailed in those papers, the study of deaf education is a highly complex line of research 
and often the normative goal is to bring the learning gains of the deaf students more in line with those seen for 
hearing students. The target subjects for the HMD study were 14-15 year old students in the United States (at about 
the 9 th grade level) and therefore we needed to establish baseline measurement for a typical hearing group of 
students at that same level. This motivates us to consider two additional questions: what do 9 th grade students 
already know before the test, and how much does their score improve over one planetarium-based teaching session? 
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METHOD 


Facilities 

The testing for this study was done in the Brigham Young University Roy den G. Derrick Planetarium on a 
college campus in the United States. This planetarium consists of a 39’ dome with 119 unidirectional seats. The 
planetarium is equipped with two projection systems: a Zeiss zkp3 star ball and an Evans and Sutherland Digistar5 
duel projector system. The Zeiss star ball mimics the relative brightness of the stars with a great deal of accuracy. It 
can project stars and deep sky objects below the visual cut-off of most people. These objects can only be seen with 
binoculars. The Evans and Sutherland Digistar5 is a full dome projection system that provides a large number of 
graphical tools that can be used with the star field. However, the relative brightness of the stars is not as accurate as 
with the Zeiss system and brightness is mostly represented by using more pixels. 

Due to the high quality star field generated by the Zeiss system it has been used to teach students in the 
introductory descriptive astronomy class a set of constellations, bright stars, and = deep sky objects. The list can be 
found on the sample constellation quiz in Appendix A. However, the impact of the projector system on the learning 
of the constellations has never been tested. As mentioned earlier, one of the purposes of this current project is to 
establish baseline measurements for the Zeiss systems that can then be used in testing the effectiveness of different 
projector systems. In addition, the head-mounted displays (HMD) detailed in Jones and colleagues (2014), and other 
HMD systems, generate their own light an impact the dark adaption of the participant. The Zeiss projector generates 
an accurate, but fainter star field. Using the HMD systems in a planetarium might require the use of the Digistar5 
system instead of the Zeiss star ball. 

Procedure 

Over three typical 16-week semesters we pre-tested non-science majoring undergraduate college students at 
the beginning of the semester by giving a survey of constellation and bright star knowledge—referred hereafter as 
the “constellation quiz”—prior to any instruction. This pre-test followed the same method for the actual 
constellation quizzes that is detailed a little later on. For two additional semesters we asked the students to give us a 
self-report of which constellations and bright stars they thought they could identify if we pointed them out in the 
planetarium. We changed to this format for the “pre-testing” phase of this project due to the lengthy amount of time 
for a traditional constellation quiz pre-test. We wanted to see if we could replace a true pre-test, with a self-report 
questionnaire that requires only a fraction of the time, relying on earlier work in this regard by Adams and Slater 
(1998). This seemed especially important for the 14-15 year old students—hereafter referred to simply as 9 th 
graders—test subjects who were taking the pre-test, receiving instruction and taking a post-test all in one visit. For 
all five semesters covered by this study we also collected some demographic data of the university students. 
Students were asked to give their gender, the type of sky they grew up under (city, suburbs, country) and how many 
constellations they had previously learned. Some students chose not to report the demographic information on the 
form and their numbers are labeled as “Not given.” These students are counted in the totals only. 

Instruction on the constellations for the university student takes place in a separate instructional lab 
learning experience held in the planetarium. Labs are normally offered about 8 times per week over the majority of 
each semester. Students are free to attend any lab, attend as often as they want, and when they feel ready, can take 
the ‘constellation quiz’ during the second half of the lab period. Previously, students generally attended the lab 3 
times before attempting the quiz. It should be noted that the “constellations” being taught are the naked eye stars 
that comprise the traditional mythological creature or object associated with the name of the constellation. The 
constellation quiz shown in Appendix A shows the name of the constellation with the common shape associated 
with that constellation given in parentheses. An example would be Ursa Major as The Great Bear. When the 
planetarium lecturing teaching assistants (TAs) point out the constellations on the planetarium dome using a red 
laser pointer, they show the naked eye stars associated with the constellation and trace out a shape that corresponds 
to the creature or object. It is very similar to doing a dot to dot with the stars, without the laser pointer leaving a line 
however. TAs will also make connections to relative positions from one constellation to another, and their 
relationships to some of the brighter stars and deep sky objects. An example could be how parts of Ursa Major can 
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be used to easily find Polaris and Arcturus. The TAs share traditional legends for many of the constellations 
including the connections between multiple constellations that belong to the same story. TAs may also add any 
anecdotal information that helps them remember the constellations. An example of this might be how Bootes, the 
Herdsman, looks more like a kite. All this instruction is done during the first half of each hour long lab session. 

Following each instruction session the students are given the constellations quiz. The TA’s would begin by 
assigning a unique number to each constellation, star, and deep-sky object. This ranged from 1 to 50. However, it 
should be noted that this was not a random assignment. To prevent the need to constantly move the sky, the 
numbers would tend to be groups by objects up in the same season. At this point the TA would point out the object 
that was assigned #1 on the quiz and tell the students this was object 1. The student would be given time to find the 
object name on the provided quiz answer sheet they thought was represented by the object pointed out by the TA 
and place a ‘1’ on the line in front of the name. The TA would then continue the same process through all 50 
objects. At the end of the quiz the TA would ask if the students wanted any of the objects pointed out again. This 
provided them a chance to change answers. One final note, while all the bright stars are part of a constellation that is 
also on the quiz, the TAs do not necessarily quiz the stars and their respective constellations at the same time. 
During the semesters of testing we allowed the students to take the quiz as many times as they wanted without 
penalty. 


In addition to surveying the knowledge of the participating university students, we also surveyed two 
groups of 14-15 year old, 9 th grade students who had come on field trips to the planetarium as a comparison data set. 
These 9 th graders were tested and instructed in an identical fashion as the university students, but were limited only 
to the constellation identification sections: Our motivation for this limit was to cut down on the overall time needed 
to acquire the data and reduce testing fatigue. For one group we gave the pre-test, taught the constellations, and then 
post-tested. For a second group we had them self-report their prior knowledge, then taught the constellations, and 
finally post-tested. The breakdown on the sample size for the two groups is listed in Table 1. 


Table 1. Numbers of Students 



Semester 1 
Pre-test 

Semester 2 
Pre-test 

Semester 3 
Pre-test 

Semester 4 
Self-Report 

Semester 5 
Self-Report 

9 th Grade 
Pre-test 

9 th Grade 
Self-Report 

Male 

48 

6 

56 

17 

111 

- 

- 

Female 

87 

20 

95 

14 

157 

- 

- 

Not given 

59 

5 

12 

0 

23 

85 

82 

Total 

194 

31 

163 

31 

291 

85 

82 


Participants 

We must start by acknowledging that the primary sample for this project was drawn from a single, 
religiously-oriented university with limited diversity. However, since this work is designed to provide a baseline we 
feel that even a limited demographic provides a starting point for future research by ourselves and other. The 
different populations tested are detailed below. 

The descriptive astronomy survey class is considered a general education class and is a possible option for 
the physical science general education distribution requirement for graduation. The class is available to all students 
regardless of year in school. Over five semesters, 710 undergraduate students participated in this study. Since the 
sample size was large we judged that it could be represented by the overall demographics of the university as a 
whole. University students come from all 50 states, the District of Columbia, and 6% of the student body come 
from other countries and are summarized in Table 2. 

We also collected data from 167 9 th grader students who visited the planetarium as part of field trips. These 
came from three different school groups. As no demographic information was collected from these groups, we will 
use the County demographic information ( http://quickfacts.census.gov/qfd/states/49/49049.html) . Once again the 
data is summarized in Table 2. 
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Table 2. Demographics of Source Population of Students 



White 

Asian/Pacific 

Islander 

Hispanic 

African- 

American/Black 

Native 

American 

Other/Multiple 

University 

84.0% 

4.0% 

2.5% 

0.6% 

0.4% 

8.5% 

County 

83.6% 

2.4% 

11.0% 

0.7% 

0.8% 

1.5% 


RESULTS 


Comparison of Data Sets of University Students 

Looking exclusively at the university data, we compared each semester to look for unexpected, systematic 
biases. In Figure 1 we show a set of boxplots for the five semesters of data for constellations. Since pre-testing 
proved to be challenging during the first three semesters of testing we decided to let the students self-report their 
knowledge. In Figure 1 the first three semesters (to the left) represent semesters when we pre-tested at the 
beginning of the semester. The last two (on the right) represent semesters in which with had the students self-report 
their knowledge. From this data it is clear that there are minimal concerns about using the data from the five 
semesters as a single data set, at least for the constellation portion of the quiz. Four of the five semesters have a 
median score of 2 out of 30 and the middle 50% for each semester was always in the range from 1 to 4 correct 
answers. This clearly shows that there is some pre-knowledge from the students and that it is consistent over 
different semesters. 

In the right hand graph of Figure 1 we show the same set of data for the stars portion of the quiz. There is a 
little bit more of a difference here than was seen with the constellations. The self-report scores seem to be a bit 
lower than those from the pre-tests. However, for both self-report semesters the median score is one, just like the 
three pre-test semesters. An average score of one is what is expected for random guessing on any matching 
problem. After an examination of the data we feel that we can again merge all five semesters of data into a common 
data set for further analysis. As we will detail later the results for stars is about at the level of guessing. It should be 
noted that Semester 5 has an unusual number of outliers. Due to a larger number of people reporting knowing none 
of the constellations or bright stars the 1 st to 3 rd quartile range is very small. This causes a greater number of points 
to be counted as outliers. Given a slightly wider middle quartile range we would likely see distributions similar to 
those for the other semesters. 


Figure 1. Data from the five semesters of university data collected. 

The constellation section of the quiz has a maximum score of 30 and the star section a maximum of 17. 



Semester 1 Semester 2 Semester 3 Semester 4 Semester 5 



Semester 1 Semester 2 Semester 3 Semester 4 Semester 5 


Our next comparison examined the results for male versus female students. In Figure 2 we show the 
breakdown for both constellations and stars. There might be a little concern here in both cases. For the 
constellations we found a slightly lower median score for the female students, but overall 75% of both male and 
female students had a score of 4 or less. The case for stars is a bit different, but we are dealing with a minimal 
amount of prior knowledge in both cases. For both male and female students the median score was a 1 out of 17. In 
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some ways we could just say that there is effectively no prior knowledge of stars for either group, which is also 
supported by Figure 1. A careful examination of the entire data string for stars shows a larger number of zeros for 
the female students, and that a significant fraction of these came from the self-report data set. This leads to a similar 
result to that discussed for Figure 1. The larger number of zeros means the range is small and therefore leads to 
more points being classified as outliers. If we include the outliers for the female students in the right graph of 
Figure 2 we would see something very similar to the male students. 


20 


15 


10 


5 


0 


Figure 2. Comparison of male and female university students. The graph to the left is for constellations 
and the one to the right is for stars. This is a combinations of both pre-test and self-reported data. 




In addition to gender we also asked the students to access the brightness of the sky where they grow up. 
For clarity the descriptions for each brightness level can be seen on the survey. There were about 600 university 
students that answered the demographic information. The breakdown of their descriptions of their sky brightness is: 
72 lived under very dark skies, 134 under mostly dark, 373 under kind of dark and only 21 under bright skies. A 
summary of the data for their constellation scores is shown in Figure 3. Once again there is only a small variation 
between the groups. The only point of interest here is in the group that reports growing up in bright skies, the 
median value for this group is 1 and the highest score is a 5, but it must be remembered this was a small sample size. 
The other three groups have median scores of 2 or 3 and the upper quartiles are 8. Since we feel that students 
effectively bring no knowledge of stars into the class we did not examine the breakdown for stars versus sky 
condition. If future work contains unusually large numbers of students from bright skies such as a large city this may 
have to be taken into account, but for this study we will combine all data together. 

Figure 3. Comparison of prior knowledge of constellations based on the brightness of sky. 
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Comparison of Data Sets of 14-15 Year Old, 9 th Grade Students 

As has been mentioned earlier, for this sample of 9 th grade students we focused solely on the constellation 
portion of the quiz. As part of our deaf education project we wanted to establish a set of baseline observations for a 
group of similar age hearing children. Due to the slower pace of communication when using ASL interpreters 
(everything is repeated, thus doubling the time) we decided to only use the constellation portion of the quiz. This is 
also important in terms of testing fatigue when we ran the full range of pre-test, instruction, and post-test. The time 
to run a sequence of pre-test, instruction, and post-test is normally more than an hour. Only one group of 9 th grade 
students was tested fully in this manner. 

When we examined the difference between pre-testing and self-report for the 9 th grade students we found 
something quite different than for the university students. Figure 4 shows the two distributions and clearly the 
students report knowing more than they truly know. The pre-test distribution has middle 50% range of the students 
knowing one to three constellations, with the highest scores only reaching 5 correct out of 30. Although the median 
score for self-report distribution isn’t too much different there is clearly a lot more confidence on the self-report. The 
middle 50% range is from one to five, with the maximum score of 10 and outliers as high as 16 out of 30. This is a 
bit worrisome since we would like to use a self-report to limit the testing time. If we examine Figure 5 we see that 
for pre-tests the university students have a bit more prior knowledge. The 9 th grade students median score of 2 
correct with a maximum score of 5. While the university students’ median score is similar the top scores are at 8, 
with a number of additional outliers. 

Figure 4. Comparison of pre-testing to self-report for a group of 9 th grade students. 

20 


15 


10 


5 


0 



Self-Report 


Figure 5. Pre-test comparison of university students versus 9 th grade students. 
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The case of the constellation quiz is an example of dependent probability or matching without replacement. 
The probability of getting a specific number correct from pure guessing is given Equation 1 (Siegrist, 2014). In this 
case P is the probability of getting a specific number, k , correct out of a total sample of n. Traditionally we have 
always looked at the constellation quiz as a single matching with 50 items. However, for the work presented in this 
paper we want to split the matching into three matching sections of constellations, stars, and other objects. We 
assume that during the pre-tests, students would be able to tell the difference between the constellations and stars 
being pointed out by the TA and would not mark a star name or the other objects when the TA was showing a 
constellation. 


P (N n = k)=^]: i k 0 tf- (i) 

As we examine the results of our testing we will break the data into groups of n=30 and n=17. In both 
cases we find that for complete guessing you would expect 37% to get 0, 37% to get 1, 18% to get 2, 6% to get 3, 
and 1.5% to get 4, with only about 0.5% chance of getting a higher score. The shapes for these curves can be seen 
graphically in Figure 6. From this we can see about how much prior knowledge the students had coming into the 
testing. We have chosen not to examine the matching for the three deep sky objects since the probability of 
guessing correctly is much higher for these three items and leads to no useful information. 


Prior Knowledge - University Students 

For the first 3 semesters, 388 University students who were taking the descriptive astronomy class were 
given the constellation quiz when they attended their first constellation lab of the semester. In Figure 6 we show 
two graphs, the first is the percentage of students who got each score for the 30 constellations and second the same 
distribution for the 17 stars. On each graph we include the expected distribution for pure guessing. It is clear from 
Figure 6 that there is at least some prior knowledge for the constellation portion of the quiz. We see three local 
maxima in the distribution at 2, 4 and between 8 and 9 correct answers. Although the bump between 8 and 9 appears 
small it represents about 46 students. For the stars we see a distribution that is much closer to what is expected from 
pure guessing. While we do note the local maxima at 4 correct in the constellation distribution we will later address 
this as likely coming from the correct guessing of the constellation of Draco. 


Figure 6. The distribution of pre-test scores for university students on both the constellation 
and star sections of the constellation quiz and the distribution predicted by guessing. 
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After the first three semesters of testing we found that the labs which included pre-testing, instruction, and 
then post-testing took a significant amount of time. Our data also showed that university students’ self-reports 
accurately represent what they know (see Figure 1). Therefore, we changed the testing format to a self-report. This 
allowed us to collect their prior knowledge without the need or time of a constellation quiz. The self-report was 
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administered the first day of class prior to seeing anything on the dome and took less than five minutes. The self- 
report instructed the students to mark those constellations, stars and objects they believed they would be able to 
name when pointed out on the planetarium dome. Self-reporting seemed more important for the high school and 
junior high aged children, as they would suffer from test fatigue and time constraints of essentially two quizzes and 
teaching even though they were limited to just the constellations. The results from the self-report of 314 university 
students are shown in Figure 7. For these graphs we have also included the pre-test score as solid squares. 

Figure 7. The distribution of scores from a self-report (triangles) compared to the pre-test results 
(boxes) for University students and the distribution predicted by guessing (dots connected by lines). 
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While the numbers are very similar between the pre-test and the self-report as stated earlier, it is possible to 
think that the student actually know a little bit more than they think, or are perhaps just good at guessing when 
taking the quiz. The second local maximum at 4 constellations is much less noticeable in the self-report. For stars it 
is even less likely that students have confidence in what they know. Again since the numbers are so small, we will 
be combining all the data sets for prior knowledge for university students. 

To further examine the data we looked at the percentage of students who got each constellation correct on 
the pre-test or reported knowing the constellation of the self-report (see Figure 8). For the pre-tests it was not 
surprising to find that Orion and Ursa Major were the constellations that most students already knew at about 70% 
and 45% correct respectively. What was the most surprising to us was Draco being the third most correctly 
identified, followed by Ursa Minor. Beyond those four constellations the remainder were only identified correctly 
10% or less of the time. Our first thought on the surprising result for Draco was reflecting students’ external 
knowledge because of a well known book and movie character from the popular young adult literature series, “Harry 
Potter.” However, when one examines the same graph for the self-report data one sees that people don’t report 
knowing Draco on the self-report. Our best guess here is that the need to trace out a serpentine shape to go between 
Ursa Major and Ursa Minor makes Draco easier to guess since it is the only snake like constellation on the list. We 
feel this explains the local maximum seen at 4 correct answers on the pre-tests. In addition, the self-report results 
show Ursa Major and Orion still being the most popular, but the confidence in knowing Ursa Minor is stronger than 
the actual identification of the constellation in the pre-tests. We do want to note that the phrase “Big Dipper” does 
not appear anywhere on the constellation quiz. It is very likely that students do know the “Big Dipper” asterism 
without knowing it is part of the constellation Ursa Major. Since Big Dipper is not a valid constellation, we did not 
include it in the study. 
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Figure 8. A breakdown by constellation of both the pre-test and self-report for university students. 
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In Figure 9 we show a similar breakdown by star. In this case the two stars which were most likely to be 
known ahead of time were Polaris and Betelgeuse. This is not an altogether surprising result. It should be noted 
however that the levels for the most recognized stars are much lower than the most recognized constellations. There 
is clearly less prior knowledge about bright stars. 


Figure 9. A breakdown by star of both the pre-test and self-report for university students. 



Since the pre-test and self-report distributions are similar we can make one combined graph for prior 
knowledge for both constellations and stars for the university students. In Figure 11 we show this combined 
distribution in diamonds along with the curve for pure guessing in circles for both constellations and stars. For the 
constellations we clearly see that the university students brought prior knowledge into the class with the majority of 
the students knowing about 2 constellations, or about one more than pure guessing. From Figures 9 and 10 above 
those two were most often Orion and Ursa Major. There is also the local maximum at about 8 constellations. This 
is an interesting result that will be addressed later when we discuss it in connection with the 9 th grade data. 
Although not shown in the graphs the two maxima in the constellation graph hold true for both male and female 
students. We wonder if these maxima would be seen in students from other schools or are a product of University 
culture. A look at the same distribution for stars shows that the distribution closely tracks a pure guessing 
distribution. For the most part students bring in very little prior knowledge of bright stars. 
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Figure 10. The combined distribution for pre-tests and self-reports (diamonds) compared to 
the distribution expected from pure guessing (circles). These distributions are for university students. 




Prior Knowledge - 9 th Grade Students 

For the 9 th grade students we get a different distribution for the pre-tests than those we saw for the 
university students. Recall that 9 th graders were only tested on the constellations. In Figure 12 we once again show 
the pre-test scores with the pure guessing distribution overlaid as well as the results of the university students. 
Clearly there is still some prior knowledge here, but without the long tail seen in the university students. In many 
ways the results for the 9 th grade students matches the majority of the university students and shows that most 
people do not learn more constellations between 9 th grade and starting a university. It is also interesting to note from 
Figure 13 that the two most known constellations are still Orion and Ursa Major. Again Draco shows up third and 
likely for the same reasons stated above. 

Figure 11. The distribution for 9 th grader pre-tests (inverted triangles) compared to the distribution of 
the university students (squares) and the distribution expected from pure guessing (circles). 
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Figure 12. A breakdown by constellation of both the pre-test and self-report for 9 th grade students. 
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Another set of 82 9 th grade students came to the planetarium and were asked to self-report which 
constellations they could locate in the night sky. The resulting distribution is given in Figure 13 (and seen in Figure 
12), along with the pre-test results and the pure guessing distribution. Taking this together with the self-report 
percentage for each constellation shown in Figure 13 it is clear the students have a great deal of confidence in what 
they think they know. These results are a little problematic for future research with secondary age subjects if time 
prevents doing an actual pre-test. It is very interesting to note that some constellations where 20% or more of the 
students report knowing got nearly a zero on the pre-test (see Gemini, Hercules, Leo). They also have a great deal 
of confidence in their knowledge of Ursa Minor, which is also not supported by the pre-test. It is always possible 
that since the self-report 9 th graders came from the same school, they have had a teacher in the past who did teach 
them more constellations than those that took the pre-test. A larger data set covering multiple schools would be 
needed to further resolve this observation. 

Figure 13. The distribution of 9 th grader scores from the self-report (triangles) and 
pre-test (inverted triangles) compared to the distribution expected from pure guessing (circles) 



In order to examine the average scores in more detail we looked at the combination of pre-test/self-report 
for university students compared to the pre-test for 9 th grade students. This was due to the fact that we feel we can’t 
trust the self-report for 9 th grade students. Looking at the distribution of scores for university students there appears 
to be two local maxima (see Figure 14), one near two and the other near eight correct answers. We fit the best 
possible combination of Gaussian distributions to the data and found one group at 2.04±0.08 and the second at 
8.23±0.23. If we treat this as a single distribution we find a combine average of 2.81±0.09. Clearly these mean 
values are different. The distribution of scores with the best double Gaussian fit is shown in Figure 15 where we see 
one peak near 2 correct answers and a second bump at approximately 8 correct answers. From this sample 664 
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students are represented by the main peak and 46 students in the higher group, or only 6.5% of the students. For the 
stars the overall university average is 1.05±0.05. Since the average score for guessing for any set of matching is 
always one, this again shows that there is little if any prior knowledge for the identification of bright stars. If there 
is a second maxima for the stars, like for the constellations, it is just too small to see. 

For the 9 th grade students the best fit was found to be a single Gaussian with a mean value of 1.79±0.13. If 
we apply a Welch /-test to the lower Gaussian of the university students compared to the 9 th grade Gaussian we find 
/-statistic of 1.62 with effectively infinite degrees of freedom. At a 95% confidence interval, with a two-tailed test, 
we cannot eliminate the possibility that these two groups are drawn from the same population. Therefore we 
conclude that the majority of university students (93.5%) started the class with knowledge of the constellations 
similar to that of 9 th graders. The remaining 6.5% of the class (46 students) represent a second population with a 
significantly higher score of 8.23 constellations, or 6 additional constellations. This group might be those who self¬ 
select into a descriptive astronomy class due to the subject material versus those who might take the class only to 
fulfil general education requirements. 

Figure 14. Gaussian fits to the constellation scores for university students (left) and 9 th graders (right). 

Note that the distribution of university students contains two separate Gaussian fits. 
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Constellations Learned in a Single Learning Experience 

A very important part of this testing is to determine how much the students learn in one session of 
instruction. If these results are to be used as a baseline for the effectiveness of a teaching method we need to know 
the results from that single session. In Figure 15 we show the score distribution for prior knowledge and the results 
after one constellation lab. For constellations more than 75% of the students had learned more than half the 
constellations in a single lab instruction session. In the case of the stars only about 50% of the students were able to 
identify half the bright stars on the quiz. Although we told the university students that we would only keep their 
highest score some didn’t try the quiz after one session of instruction, leaving their quiz blank. Those students who 
left the quiz completely blank during quizzes after instruction have been removed from the sample. It should be 
noted at this point that the quiz in all cases was given immediately after instruction. Future work is planned to 
explore long term retention. 

The scores for 9 th graders, shown in Figure 16, show a wider range of scores on the constellations with less 
than half the students getting half the constellations correct after one lab. Some of this may be the fact that the 
university students were doing this for a grade and therefore had more incentive to do better. However, the data 
from the 9 th grade students gives us a great baseline for studies involving school groups coming to the planetarium. 
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Figure 15. Comparison of Pre-test/Self-report values to results after 1 teaching session for university students. 
The left graph is for constellations and the right is for stars. Maximum score for constellations is 30 and for stars 17. 
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Figure 16. Comparison of Pre-test values to results after 1 teaching session for 9th students (Constellations Only). 
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Constellation Learning Curve over Multiple Learning Experiences 

Over five semesters we were able to track the scores of university students over multiple instructions and 
quizzes. We did not track the scores of individual students only the scores and the number of lab sessions attended. 
During these semesters students were allowed to take the quiz as many times as they wanted until they received a 
score they wanted to keep. Therefore, we were able to track the learning curve for the constellations, stars, and 
other objects. In Figure 17 we can see that a great deal of the learning takes place in the first lab session. However, 
from the first two labs we still see students who are getting zero correct, even after removing the zeros from students 
who didn’t try as evidenced by a blank quiz. By the time we reach 3 lab sessions we find that the vast majority of the 
students are getting full points on the constellations and by 4 labs only a few people do not get 30 out of 30. It 
should be pointed out that over many years of giving the constellation quiz the faculty told students that they should 
be able to get high scores after 2 to 3 labs. It turns out that this was a reasonable statement. 

Looking at the individual constellations we can see that some constellations appear harder to learn than 
others. In Figure 18 we show the percentage of students who got each constellation correct on the pre-test/self- 
report, after 1 lab, 2 labs, and 3 labs. Orion and Ursa Major quickly rise to over 90% correct, with Draco not far 
behind. Aquarius seems to be the hardest to learn if we track the pattern out to 5 labs sessions. Perhaps the 
strangest pattern on learning the constellation appears if we look all the way to 6 labs. Sagittarius reaches 92% 
correct by the 3 rd lab, but then the percentage of students getting it correct drops again. This could be either a 
retention issue or just that the people who need to take a quiz more times had a hard time learning this constellation. 
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Figure 17. Learning curve for constellations from pre-test to 6 sets of instruction. 
Maximum score on this section of the quiz is 30. 
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Figure 18. Percentage correct for each constellation from Pre-test to 3 laboratory sessions. 
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In Figure 19 we show the same learning curve for the stars. Although similar in trend to the constellation 
portion of the quiz, the learning is clearly a little slower for the stars. We see a wider range after 1 lab, with a fair 
fraction of the students still getting less than 50% of the stars correct. Beyond that point the pattern is similar to the 
constellations. However, we do note the interesting result for those students who attended 6 labs. At 5 labs almost 
everyone got perfect scores on the stars, but those who needed to take it a 6 th time reverted to missing more stars. 
We don’t have an explanation at this point for this result, but we note that only 5 students took the quiz 6 times. 
Figure 20 shows that by 3 labs the students aren’t doing as well on learning the stars. A larger fraction of the stars 
are not being identified correctly. Many stars have scores in the 80-90% ranges, while the majority of the 
constellations are already in the 90% range. 
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Figure 19. Learning curve for stars from pre-test to 6 sets of instruction. Maximum score on this section of the quiz is 17. 
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Figure 20. Percentage correct for each star from pre-test to 3 laboratory sessions. 



DISCUSSION 

From an examination of university students who registered for the basic descriptive astronomy class we 
found that the majority of students began class knowing about the same number of constellations as 9 th grade 
students. This represented 93.5% of the students or about 664 students out of group of 710. Those students could 
mostly identify Orion and Ursa Major, which is consistent with their average score being 2.04±0.08. The university 
students do come from all 50 states and other nations, but the 9 th graders are all from Utah. It would be intriguing to 
see if this same familiarity extends to other regions of the country. Steps are underway to extend these 
measurements to other locales. As far as the extent of this study it appears that the two constellations are learned 
prior to 9 th grade. In the case of knowledge of stars we only have the university students. Of the 17 stars on the quiz 
there were none that were known by more than 50% of the students on the pre-test/self-report. From the 
combination of pre-test and self-report we find that Polaris is known about 35% of the time and Betelgeuse around 
18% of the time. The average score for the stars section of the quiz is 1.05±0.05, or almost exactly 1. This is the 
score that would be expected for guessing. However, we assume that if the students are guessing, no one star would 
be known more than any other. But two stars, Polaris and Betelgeuse, are clearly marked more often inferring that 
university students do know one star, and are not all just guessing. In both situations there is very little knowledge 
ahead of time of constellations or bright stars. The low numbers of known constellations and bright stars allow that 
constellations and bright stars can be very useful content to study teaching methods in a planetarium setting. In 
addition, we have shown that university students can be trusted to report their knowledge of constellations and stars 
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eliminating the lengthy pre-testing. In contrast, the 9 th graders must be pre-tested. We have clearly laid out a baseline 
for future work here at BYU. Extending the baseline to other geographic locales will help see if there are any biases 
due to a single site. 

For the university group we also saw a second group of students who had an average score of 8.23±0.23. 
This group is found in both the pre-test and self-report, therefore we feel this is truly a second population. This 
group represents 6.5% of the students in our classes, or 46 students. Likely we are seeing that 93.5% are taking the 
descriptive astronomy survey class just for general education distribution credit, while the 6.5% already had a 
deeper interest in astronomy. Testing of students not signed up for astronomy will help see if this is indicative of the 
population in general or just those who sign up for an astronomy class. 

The learning curve is also interesting. As was explained, the students were taught the constellations using a 
red laser pointer to show the naked eye stars as well as told stories from the myths and legends. Following 
instruction they were immediately tested. After only one instruction, the median score for constellations went up 
from 2 to 25 out of 30 for university students and to 14 for 9 th graders. Is this the preferred method for learning? Do 
the stories play any role? Is there any additional way to improve learning? Knowing now that there is a low number 
of known constellations, we can set up future studies to answer these very questions. The BYU planetarium also has 
a Digistar5 projection system that will enable us to project an illustration over the bright stars of the typical character 
associated with the constellation, the name of the constellation, and the naked eye stars connected by lines to create 
a stick figure type illustration. Now that we have a baseline we can see if there are better methods for instruction. 
We would also like to check these methods for long term retention. 

While we did allow the students to take the quiz as often as they wanted, it was noted, however, that some 
did not take it each time they were taught the constellations. This was seen in the fact that the data set after two labs 
was larger than the data set for one lab and that some quizzes were turned in blank. It would be intriguing to study 
the effect of actually taking the quiz versus not taking it. Brown and colleagues (2014) have emphasized the 
importance of retrieval or testing in the learning process. Do students learn the constellations and stars faster or for 
longer retention by taking the constellation quiz after every learning session? This would need to be more strictly 
enforced in order to make any conclusions. It was seen that the university students had a steeper learning curve after 
one session than the 9 th graders. There are many possible reasons for such a result, the most likely being the fact that 
the university students were doing this for a grade. This could be a fruitful topic for future study. 

In addition there are a number of logical questions that arise. Since the distributions for the 9 th grade 
students and the majority of university students are the same, when did they learn the first two constellations? 
Whenever they learned Orion and Ursa Major, it seems that a lot of them retained that information. How well do the 
university students retain the constellations and bright stars they learned as part of the descriptive astronomy class? 
This is the next step in our study here at our institution. 

We have conducted a study to establish baseline measurements of the amount of prior-knowledge 
university students and 9 th graders have of constellations and bright stars in the sky. We have found that for both 
groups about 50% of the students knew approximately two constellations (Ursa Major and Orion). For the pre-test 
portion of the test we found that the constellation of Draco was identified correctly the third most often. However, 
the self-reports didn’t suggest that the students actually knew the constellation. More likely the need to draw a 
serpentine shape when pointing out Draco made it the easiest constellation to guess correctly. Perhaps the most 
interesting finding was that the majority of college students scored exactly the same as the 9 th grade students. They 
hadn’t learned any additional constellations between 9 th grade and the university class. However, there was a small 
group representing 6.5% of the students that scored significantly higher. It would be interesting to examine the 
source of this group. On the university portion of the test we found that about 35% of the students knew one bright 
star (Polaris or Betelgeuse), but the overall test scores were the same as expected for pure guessing. 

In addition to setting the baselines for prior knowledge we also examined the learning curve for university 
students over multiple laboratory sessions in the planetarium. For the constellations we found that after one lab 75% 
of the students scored 18, or higher, out of 30. More than half the students got at least 24 out of 30. After three lab 
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sessions 75% of the students were scoring 28 or higher out of 30. This matches what faculty had anecdotally told 
students for many years when asked how many labs students should attend. For the 9 th grade group we found that 
after one session the score distribution was much lower than for the university students. The fact that the university 
students had a grade associated with their score might have impacted the effort put into learning during that first 
session. 


Establishing a baseline set of data for learning constellations opens a large number of additional projects. 
The work presented here was completely done using the Zeiss star ball, which generates a very accurate star field. 
Would the results be different if we used a full dome graphics system such as Digistar5? With the Digistar5 we 
could also project the official constellations boundaries. Would doing this improve the scores after one lab? Will we 
see significantly different results if the testing was done on a different demographic group? Perhaps one of the most 
important questions that still needs to be addressed is how well does the knowledge transfer to the real sky (viz., 
Smolinski, 2014)? 


ACKNOWLEDGMENTS 

We would like to acknowledge support for this work from a National Science Foundation Grant (IIS-1124548) and 
from the Sorensen Impact Foundation. We would also like to thank all astronomy faculty at for allowing us to test 
their students and all the course teaching assistants who administered the constellation quiz. 


AUTHOR INFORMATION 

Dr. Eric Hintz is an associate professor in the Department of Physics and Astronomy at Brigham Young University. 
His education research focuses primarily on the impact of teaching astronomy in a planetarium from the junior high 
to undergraduate levels. E-mail: hintz@physics.byu.edu 

Prof. Maureen Hintz is a part-time instructor of physics and astronomy at both Brigham Young University and 
Utah Valley University. 

Dr. M. Jeannette Lawler is the planetarium director for the Department of Physics and Astronomy at Brigham 
Young University. As well as the coordinator for the general education physical science class. 

REFERENCES 

Adams, J. P., & Slater, T. F. (1998). Using action research to bring the large class down to size. Journal of College 
Science Teaching, 28(2), 87. 

Baily, J. M. & Slater, T. F. (2004). A review of astronomy education research. Astronomy Education Review, 2(2), 
20-45. 

Brazzelli, B., & Espinoza, S. (2009). Meta-analysis of planetarium efficacy research. Astronomy Education Review, 

S( i). 

Brown, P.C., Roediger, H.L., & McDaniel, M.A. (2014). Make it Stick. Cambridge, MA: The Belknap Press of 
Harvard University Press. 

Eriksson, U., Linder, C., Airey, J., & Redfors, A. (2015). Who needs 3D when the Universe is flat? Science 
Education, 93 (3), 412-442. 

Hintz, E. G., Jones, M. D., Lawler, M. J., Bench, N., & Mangrubang, F. (2015). Adoption of ASL classifiers as 
delivered by head-mounted displays in a planetarium show. Journal of Astronomy and Earth Science 
Education, 2(1), 1-16. 

Jacobson, J. (2013). Digital dome versus desktop display: Learning outcome assessments by domain experts. 

International Journal of Virtual and Personal Learning Environments, 4(3), 51-65. 

Jones, M., Lawler J., Hintz E., Bench N., Mangrubang F., & M. Trullender, (2014). Head mounted displays and deaf 
children: Facilitating sign language in challenging learning environments. Interaction Design and 
Children, Aarhus, Denmark (IDC, 2014). 


Copyright by author(s); CC-BY 


132 


The Clute Institute 




Journal of Astronomy & Earth Sciences Education - December 2015 


Volume 2, Number 2 


Mallon, G. L., & Bruce, M. H. (1982). Student achievement and attitudes in astronomy: An experimental 
comparison of two planetarium programs. Journal of Research in Science Teaching, 19, 53. 

Plummer, J. D. (2009). Early elementary students’ development of astronomy concepts in the planetarium. Journal 
of Research in Science Teaching, 46 (2), 192-209. 

Plummer, J. D., Kocareli, A., & Slagle, C. (2014). Learning to explain astronomy across moving frames of 

reference: Exploring the role of classroom and planetarium-based instructional contexts. International 
Journal of Science Education, 36(1 ), 1083-1106. 

Reed, G. (1970). Is the planetarium a more effective teaching device than the combination of the classroom 
chalkboard and celestial globe? School Science and Mathematics, 70(6), 487-492. 

Ridky, R. W. (1975). The mystique effect of the planetarium. School Science and Mathematics , 75(6), 505-508. 

Siegrist, K. (2014) The Matching Problem. Retrieved April 15, 2014, from 
http://www.math.uah.edu/stat/um/Matching.html 

Slater, T. F. (1993). The Effectiveness of a Constructivist Epistemological Approach to the Astronomy Education of 
Elementary and Middle Level In-Level Teachers (Ph.D. dissertation, University of South Carolina). 

Smolinski, J. P. (2014). Tablet computing devices to bridge the gap between planetarium and night sky. Bulletins of 
the American Astronomical Society, 445, 14 . 

Sunal, D. W. (1976). Analysis of research on the educational uses of a planetarium. Journal of Research in Science 
Teaching, 13(4), 345-349. 

Turk, C., & Kalkan, H. (2015). The effect of planetariums on teaching specific astronomy concepts. Journal of 
Science Education and Technology, 24, 1-15. 

Yu, K. C., Sahami, K., Sahami, V., & Sessions, L. C. (2015). Using a digital planetarium for teaching seasons to 
undergraduates, Journal of Astronomy and Earth Science Education, 2(1), 33-50. 

Zimmerman, L., Spillane, S., Reiff, P. & Sumners, C. (2014). Comparison of student learning about space in 

immersive and computer environments. Journal Review of Astronomical Education Outreach, 1(1), A5- 
A20. 


Copyright by author(s); CC-BY 


133 


The Clute Institute 




Journal of Astronomy & Earth Sciences Education - December 2015 


Volume 2, Number 2 


APPENDIX A 
CONSTELLATION QUIZ 

Circle Class: Section 1 Section 2 Section 3 Section 4 

How many lab sessions have you attended (including today)?_ 

Name_ 

Constellations: 


Andromeda (Princess) 
Aquarius (The Water Bearer) 
Aquila (The Eagle) 

Aries (The Ram) 

Auriga (The Charioteer) 
Bootes (The Herdsman) 
Cancer (The Crab) 

Canis Major (The Great Dog) 
Canis Minor (The Little Dog) 
Capricorn (The Sea Goat) 
Cassiopeia (Queen) 

Corona Borealis (The Crown) 
Corvus (The Crow) 

Cygnus (The Swan) 

Draco (The Dragon) 

Gemini (The Twins) 

Hercules (Strong Man) 

Leo (The Lion) 

Libra (The Scales) 

Lyra (The Lyre or Harp) 
Orion (The Hunter) 

Pegasus (The Winged Horse) 
Perseus (Running Man) 

Pisces (The Fish) 

Sagittarius (The Teapot) 
Scorpius (The Scorpion) 
Taurus (The Bull) 


Ursa Major (The Great Bear) 
Ursa Minor (The Little Bear) 
Virgo (The Young Woman) 


Stars: 

_Aldebaran 

_Algol 

_Altair 

_Antares 

_Arcturus 

_Betelgeuse 

_Capella 

_Castor 

_Deneb 

_Polaris 

_Pollux 

_ Procyon 

_Regulus 

_Rigel 

_Sirius 

_Spica 

_Vega 


Other: 

_Pleiades (The Seven Sisters) 

_Hyades 

_Orion Nebula (M42) 


Answer the questions to the best of your knowledge. This part of the survey is optional and will not affect your 
grades. 

1. Gender: 

Male □ Female □ 

2. Which Hemisphere of the earth have you spent the majority of your life? 

Northern □ Southern □ 

3. On average, how dark was the night sky where you lived? Circle one. 

a. Very dark, could see all stars (out in the country) 

b. Mostly dark, could see a lot of stars (city outskirts) 

c. Kind of dark, could see some stars (suburbs) 

d. Not dark, could see no stars (city) 

4. How many constellations have you learned before beginning this class? Circle one. 

a. None b. 1-2 c. 3-5 d. 5-10 e. 10 or more 
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